Super resolution methods for electro-optical systems

ABSTRACT

In an optical system having a detector means and processor means in which imaging data is obtained comprising noisy blurred scene data containing an object to be reconstructed, and noisy blurred background data of the same scene, a method for increasing the spatial resolution of the imaging data produced by the optical system, comprising the steps of converting the imaging data into a first matrix, regularizing the first matrix by performing nth order Tikhonov regularization to the first matrix to provide a regularized pseudo-inverse (RPI) matrix and applying the RPI matrix to the first matrix to provide a reconstructed image of the object.

RELATED APPLICATIONS

The present application is a divisional of Application Ser. No.09/081,842, now U.S. Pat. No. 6,295,392 filed on May 20, 1998.

The present application is related to copending commonly assigned U.S.patent application, Ser. No. 08/763,610, filed on Dec. 11, 1996,entitled “Apparatus and Method For Providing Optical Sensors With SuperResolution”, incorporated herein by reference.

FIELD OF THE INVENTION

The present invention relates generally to optical devices, and moreparticularly to an optical sensor utilizing background reconstructionimage processing techniques in order to provide a much higher level ofresolution of a localized object within a background scene.

BACKGROUND OF INVENTION

Optical sensors are devices which for decades have been used to detectand record optical images. Various types of optical sensors have beendeveloped which work in the Ultra Violet Bands, Infra Red Bands as wellas in the Visible Bands of operation. Examples of such devices includeWeather Sensors, Terrain Mapping Sensors, Surveillance Sensors, MedicalProbes, Telescopes and Television Cameras.

An optical sensor typically includes an optical system and one or moredetectors. The optical system portion is made up of various combinationsof lenses, mirrors and filters used to focus light onto a focal planelocated at the image plane of the optical system. The detectors whichmake up the image plane are used to convert the light received from theoptical system into electrical signals. Some types of optical sensorsuse film rather than detectors to record the images. In this case, thegrain size of the film is analogous to the detectors described above

An important performance characteristic of optical sensors is their“spatial resolution” which is the size of the smallest object that canbe resolved in the image or, equivalently, the ability to differentiatebetween closely spaced objects. If the optical system is free fromoptical aberrations (which means is being “well corrected”) the spatialresolution is ultimately limited by either diffraction effects or thesize of the detector.

Diffraction is a well known characteristic of light which describes howlight passes through an aperture of an optical system. Diffractioncauses the light passing through an aperture to spread out so that thepoint light sources of a scene end up as a pattern of light (known as adiffraction pattern) diffused across the image. For a well corrected,unobscured optical system known as a diffraction limited system, thediffraction pattern includes a very bright central spot, surrounded bysomewhat fainter bright and dark rings which gradually fade away as thedistance from the central spot increases.

An optical sensor that is designed to be diffraction limited usually hasa very well corrected optical system and detectors sized so that thecentral spot of the diffraction pattern just fits within the active areaof the detector. With conventional sensors, making the detectors smallerdoes not improve resolution and considerably increases the cost due tothe expense of the extra detectors and the associated electronics.

The size of the aperture used in the optical system determines theamount of resolution lost to diffraction effects. In applications suchas camera lenses and telescope objectives, the aperture size is normallyexpressed as an f-number which is the ratio of the effective focallength to the size of the clear aperture. In applications such asmicroscope objectives, the aperture size is normally expressed as aNumerical aperture (NA) which is the index of refraction times the sineof the half angle of the cone of illumination. For a given focal length,a high f-number corresponds to a smaller aperture, while a higherNumerical aperture corresponds to a larger aperture.

A basic limitation of conventional optical sensors is the aperture sizerequired for a given level of resolution. Higher resolution imagesrequire larger apertures. In many situations the use of such a system isvery costly. This is because using a larger aperture requires asignificantly larger optical system. The cost for larger systems whichhave apertures with diameters greater than one foot is typicallyproportional to the diameter of the aperture raised to a power of “x”.The variable “x” usually ranges from 2.1 to 2.9 depending on a number ofother particulars associated with the sensor such as its wave band,field of regard, and field of view.

The size of the optical sensor is particularly relevant for systems thatfly on some type of platform, either in space or in the air. Under suchconditions the sensor must be light weight, strong, and capable ofsurviving the rigors of the flight environment. Thus the cost of goingto a larger optical system can be in the hundreds of millions of dollarsfor some of the larger and more sophisticated sensors. Practicalconsiderations, such as the amount of weight the host rocket, plane,balloon, or other vehicle can accommodate, or the amount of spaceavailable, may also limit the size of the sensor. These practicalconsiderations can prevent a larger system from being implemented nomatter how large the budget.

A number of optical imaging techniques have been developed to increasespatial resolution One such technique is known as sub-pixel resolution.In sub-pixel resolution the optical system is limited in spatialresolution not by diffraction but by the size of the detectors orpixels. In this case, the diffraction pattern of the aperture is muchsmaller than the detectors, so the detectors do not record all theresolution inherent in the optical system's image. Sub-pixel resolutionattempts to reconstruct an image that includes the higher resolution notrecorded by the detectors. This technique does not require hardware orsystem operation changes in order to work. Examples of sub-pixelresolution techniques are disclosed in an article in ADVANCED SIGNALPROCESSING ALGORITHMS, ARCHITECTURES AND IMPLEMENTATIONS II, by J. B.Abbiss et al., The International Society For Optical Engineering, Volume1566, P. 365 (1991).

Another example is the use of “thinned aperture” systems where forexample, a widely-spaced pattern of small holes is used as a substitutefor the complete aperture. However, even “thinned a pertures” arelimited in resolution by diffraction theory and by the outer diameter ofthe widely-spaced pattern of small holes. Note that currentelectro-optical systems are sometimes designed so that the size of theirdetector matches the size of the diffraction blur of their optics.

Other examples of optical imaging techniques are disclosed in an articleentitled SUPER RESOLUTION ALGORITHMS FOR A MODIFIED HOPFIELD NEURALNETWORK, by J. B. Abbiss et al., IEEE Transactions On Signal Processing,Vol. 39, No. 7, July 1991 and in a paper entitled FAST REGULARIZEDDECONVOLUTION IN OPTICS AND RADARS, by J. B. Abbiss et al., presented atthe 3rd IMA Conference on Mathematics in Signal Processing. Thesetechniques use linear algebra and matrix techniques to restore signalsor images from a limited discrete data set.

The previously described techniques have a number of drawbacks withregard to optical sensors. First, only one of these techniques isdirected toward a diffraction limited device. In addition, thesetechniques often produce systems of equations which cannot be solved dueto the practical constraints on computing power. Furthermore, none ofthe previously described techniques specify either the types ofdetectors or other system parameters which are used along with thesetechniques.

It is, therefore, an object of the present invention to provide atechnique for optimally increasing the resolution of an optical sensorthat is diffraction limited (using super-resolution techniques) withoutusing a substantially larger aperture.

SUMMARY OF THE INVENTION

The invention provides a method of improving the spatial resolution ofan object using a background reconstruction approach wherein a localizedobject containing high spatial frequencies is assumed to exist inside abackground scene containing primarily low and/or very high spatialfrequencies compared to the spatial frequencies of the localized object.The imaging system cannot pass these high spatial frequencies (neitherthe high frequencies of the object, nor the very high frequencies of thebackground). The background image's low spatial frequencies are used toreconstruct the background scene in which the localized object issituated. Using this reconstructed background and the space limitednature of the localized object (i.e. it is only present in part of thescene, not the entire scene), the high spatial frequencies that did notpass through the optical system are restored, reconstructing a detailedimage of the localized object.

The improvement comprises filtering the noisy blurred background data ofthe same scene to obtain noise suppressed data; applying estimations ofpoint spread functions associated with the noise suppressed data andoptical system to estimates of the noise suppressed data to obtain areconstructed background image (I_(r)(X)); and low pass filtering thenoisy blurred scene data containing the object to be reconstructed (D1)and using the reconstructed background image (I_(r)(x)) to eliminate thebackground data from the image data to obtain a reconstructed image ofan object with increased spatial resolution.

BRIEF DESCRIPTION OF THE DRAWINGS

The above objects, further features and advantages of the presentinvention are described in detail below in conjunction with thedrawings, of which;

FIG. 1 is a general block diagram of the optical sensor according to thepresent invention;

FIG. 2 is a diagram of a linear detector array tailored to the presentinvention;

FIG. 3 is a diagram of a matrix detector array tailored to the presentinvention;

FIG. 4 is a diagram illustrating the size of an individual detectortailored to the present invention;

FIG. 5 is a diagram illustrating how the detector grid is sized withrespect to the central diffraction lobe;

FIG. 6 is a diagram of a multi-linear detector array tailored to thepresent invention;

FIG. 7 is a diagram of another version of a multi-linear detector arraytailored to the present invention;

FIG. 8 is diagram illustrating the operation of a beam splitter as partof an optical system tailored to the present invention;

FIGS. 9A-B represents an image scene and a flow diagram of thenon-linear background reconstruction technique of the image sceneaccording to the present invention;

FIG. 10 is a diagram illustrating the Richardson-Lucy backgroundreconstruction portion of the non-linear image processing techniqueaccording to the present invention;

FIG. 11 is a diagram illustrating the Object Reconstruction andbackground subtraction portion of the non-linear image processingaccording to the present invention;

FIG. 12 is a schematic illustrating the Fourier Transform characteristicof binotf.

FIG. 13 is a schematic of the relationship of binmap to the object ofinterest within a window of a particular scene according to the presentinvention; and

FIGS. 14A-D show the effects of super-resolution of simulated bartargets using the non-linear reconstruction technique according to thepresent invention;

FIGS. 15A-D show the Fourier analysis of simulated bar targetsassociated with FIGS. 14A-D;

FIGS. 16A-C show simulated images associated with a thinned apertureoptical system configuration using non-linear method.

FIGS. 17A-C show the super-resolution of an image of a figure taken froma CCD camera and reconstructed using the non-linear method according tothe invention;

FIGS. 17D-F show the Fourier image analysis of the image of FIGS. 17A-C;

FIGS. 18A-B show the Fourier analysis of superimposed truth, blurred,and reconstructed images as function of frequency for SNR values of 50and 100, respectively;

FIG. 19 is a flow diagram of the linear algebra backgroundreconstruction technique according to the present invention.

DETAILED DESCRIPTION OF THE INVENTION

The present invention is directed to a method of super-resolution of animage for achieving resolutions beyond the diffraction limit. This isaccomplished by a method of background reconstruction, wherein alocalized object containing high spatial frequencies is assumed to existinside a background scene containing primarily low and/or very highspatial frequencies compared to the spatial frequencies of the localizedobject. The point spread function (PSF) of an optical system causes thebackground data to blend with and flow over objects of interest, therebycontaminating the boundaries of these objects and making it exceedinglydifficult to distinguish these objects from the background data.However, if one knows or can derive the background, then the object maybe disentangled from the background such that there will existuncontaminated boundaries between the object and the background scene.Another way of describing the effect of a PSF is to note that thespatial frequencies associated with the object of interest which liebeyond the optical system cutoff frequency are lost. In the backgroundreconstruction approach, a localized object such as a tractor havinghigh spatial frequencies is super-resolved inside a background scenecontaining primarily low and/or ultra-high spatial frequencies such as acornfield. Typically, the imaging system cannot pass either the highspatial frequencies of the object or the ultra-high spatial frequenciesof the background. This invention uses the background images' lowspatial frequencies to reconstruct the background scene in which thelocalized object is situated. This reconstructed background and thespace limited nature of the localized object (that is, the object ispresent in only part of the scene rather than the entire scene) can beused to restore the high spatial frequencies that did not pass throughthe optical system, thereby reconstructing a detailed image of thelocalized object.

In accordance with alternative embodiments of the present invention,both linear and nonlinear methods for reconstructing localized objectsand backgrounds to produce super-resolved images are described.

Referring to FIG. 1, there is shown a general block diagram of anoptical sensor accommodating the present invention. The sensor (10) asin conventional devices includes an optical system (12) and detectors(14). The optical system (12) which includes various combinations oflenses, mirrors and filters depending on the type of application is usedto focus light onto a focal plane where the detectors (14) are located.The optical system (12) also includes a predetermined aperture sizecorresponding to a particular Numeral aperture (NA) which, inconventional diffraction-limited devices, limits the amount of spatialresolution that is attainable. This is, as previously described, due todiffraction blurring effects.

The optical system (12) can be described by an optical transfer function(OTF) which represents the complete image forming system, and which canbe used to characterize that system.

The detectors (14) convert the light received from the optical system(12) into the electrical signals which become the data used to generateimages. In conventional sensors the detectors are configured in a lineararray for Scanning Systems or in a matrix array for Staring systems. InScanning systems, the detector linear array is swept in a directionperpendicular to the length of the array generating data one scan lineat a time with each line corresponding to one line of the image. InStaring systems the matrix array is not moved and generates all of theimaging data simultaneously. Thus each detector of the matrix arraycorresponds to one pixel of the image. It is intended that the detectors(14) of the present invention will be configured as a linear array or amatrix array depending on the type of system being used.

The detectors (14) take many different forms depending on the wavelengthof light used by the present invention. For example, in the ultravioletand X-Ray ranges such detectors as semitransparent photocathodes andopaque photocathodes can be used. In the visible range such detectors asvacuum phototubes, photomultipliers, photoconductors, and photodiodescan be used. In the infrared range, such detectors as photoconductors,photodiodes, pyroelectric, photon drag and golay cell devices can beused.

In the present invention various elements of the sensor (10) must beoptimized to be used with a particular image-processing technique. Thetype of optimization depends on the image-processing technique. As willbe described in detail later, the present invention includes twoalternative image-processing techniques. Each of these twosuper-resolution methods may be used with the sensor configurations asdescribed below.

In case one, the mentor (10) must include detectors (14) that have an“instantaneous field of view” that is equal to or less than the desiredlevel of spatial resolution. If, for example, the required resolution isone meter or less then the “instantaneous field of view” of thedetectors must be one meter or less (even though the central lobe of thediffraction pattern is much larger). This makes the pixel size of theimage produced by the sensor (10) smaller than the central diffractionlobe. (Note that, such a configuration adds additional cost to thesensors. However, for large systems the increase in cost is less thanthe cost of a larger aperture.)

The sensor (10) can obey this rule in one of two ways. One way is to usemore smaller-size detectors (14). In conventional sensors the number ofdetectors used varies anywhere from one to millions depending on theapplication.

In one embodiment of the present invention at least five timers moredetectors (14) than normal are required to achieve the desiredresolution. A diagram of a linear detector array to be used with thepresent invention is shown in FIG. 2, while a diagram of a matrixdetector array to be used with the present invention is shown in FIG. 3.The number of detectors (14) included in these arrays (28), (30) dependson the application. However, as previously pointed out to achieve thehigher resolution these arrays (28), (38) will include at least fivetimes more detectors (14) than conventional sensors for a givenapplication.

In conventional sensors, the size of the individual detector is neversmaller than the size of the central diffraction lobe. This is becauseutilizing smaller sensors serves no purpose since the resolution islimited by the optical aperture. In the present invention, the size ofthe individual detectors (14) must be smaller than the size of thecentral diffraction lobe (18), as shown in FIG. 4.

Another way of configuring the sensor (10) according to Case one isagain to use a larger number of detectors (14), but instead of usingsmaller detectors configure the optical system (12) so that more thanone detector is spread across the central diffraction lobe. This allowsconventional size detectors (14) to be used. The number of detectors(14) used again mu at be five times or more than required inconventional sensors. In order to configure the optical system (12) asdescribed above the back focal length must be adjusted so that five ormore detectors (14) spread across the central diffraction lobe (18), asshown in FIG. 5.

In Scanning systems, it is difficult to generate multiple image data byviewing the object at different times. This is because the opticalsystem assumes that the object remains stationary while being scanned.The solution is to use detectors (14) configured in a multi-linear arrayas shown in FIG. 6. The multi-linear array (20) includes a number ofindividual linear arrays (22) shifted parallel to each other by adistance (d) which is a fraction of a pixel. When the array (20) isswept each linear array (22) generates imaging data corresponding to onescan line of each of the images. Thus each linear array (22) creates oneof the images being produced. In the preferred configuration, the array(20) includes ten or more individual linear arrays 22 which are capableof producing ten or more different images.

Now two more examples of this technique are discussed. These examplesare labeled Case three and Case four.

In Case three, the sensor (10) is configured to take images of objectsin multiple color bands. In Scanning systems, this is accomplished byutilizing a multi-linear array, as shown in FIG. 7. The multi-lineararray (24) also includes a number of individual linear arrays (22)arranged in a parallel configuration shifted parallel to each other by adistance (d) which is a fraction of a pixel. A color filter (26) isdisposed over each of the linear arrays (22). The color filters (26) areconfigured to pass only a particular portion of the color spectrum whichmay include multiple wavelengths of visible light. When the array (24)is swept, each linear array (22) produces images of the same object indifferent color bands. The filters (26) are fabricated by depositingoptical coatings on transparent substrates, which are then placed overeach of the linear arrays (22). This process is well known.

In Staring systems, the multiple color-band data is created byincorporating a beam splitter in the optical system (12) and usingmultiple detector arrays. Such a configuration is illustrated in FIG. 8.The beam splitter (28) splits the incoming light (32) into multiplebeams (34); FIG. 8 shows the basic idea using a two-beam system. Due tothe operation of the beam splitter (28) each light beam (34) includes adifferent part of the color spectrum which may include one or moredifferent bands of visible or infrared light. Each light beam isdirected to one of the detector arrays (30) producing images of the sameobject in different color bands.

In Case four, the sensor (10) is a combination of the three previouslydescribed cases. This is accomplished by combining the principlesdiscussed above with regard to Cases one, two or three. In all threecases the sensor must be designed to have a signal to noise ratio whichis high as possible. This is done either by increasing the integrationtime of the detectors (14) or by slowing down the scan speed as much aspossible for scanning systems. For Case two, the system's design, or itsoperation mode, or both, are changed in order to take the requiredmultiple images in a known pattern displaced by a known distance that isnot a multiple of a pixel, but rather is a multiple of a pixel plus aknown fraction of a pixel.

Referring back to FIG. 1, coupled to the detectors (14) is a processor(16) which processes the image data to achieve the higher resolution.This done by recovering “lost” information from the image data. Eventhough the diffraction blur destroys the required spatial resolution,some of the “lost” spatial information still exists spread across thefocal plane. The small-size detectors described above are used to sampleat a 5 to 10 times higher rate than is customary in these sorts ofoptical systems in conjunction with processing enabling much of this“lost” information to be recovered and thus restoring the image to ahigher level of resolution than classical diffraction theory wouldallow.

The processor (16) uses one of two image processing techniques aNon-linear Reconstruction method using a modified Richardson-LucyEnhancement technique, and a background reconstruction approach using alinear algebra technique.

One reasonable extension of the previously described imaging techniquesis to use phase retrieval or wave front phase information to reconstructthe image and thus achieve higher resolution. Another reasonableextension of the previously described technique is to use priorknowledge of the background scene to help resolve objects that haverecently moved into the scene. The processor (16) in addition to usingone of the above described primary data processing techniques, also usesother techniques to process further the imaging data. This furtherprocessing is accomplished by standard image enhancement techniqueswhich can be used to improve the reconstructed image. Such techniquesinclude, but are not limited to, edge sharpening, contrast stretching orother contrast enhancement techniques.

The Non-linear Background Reconstruction method using a modifiedRichardson-Lucy Enhancement Technique is described as follows. In FIG.9A, there is shown a scene (10) comprising a localized object (20) suchas a tractor within a noisy blurred background (30). In FIG. 9B, inputdata D2 (Block 20), representing the noisy blurred background data, isinput into module 30 to remove the noise from D2 using a modifiedversion of the method of sieves. Note that the input data D1 and D2indicated in block 20 has been sampled at the Nyquist rate (S times thecustomary image sampling rate) and preferably at twice the Nyquist rate(ten times the customary sampling rate) to obtain robust input data.

The modified method of sieves removes noise by averaging adjacent pixelsof the noisy blurred background data D2 of the same scene together,using two and three pixel wide point spread functions. Array h₀ inequation (9 a) of module 30 represents the optical system PSF, which isthe Fourier transform of the OTF input.

The output of module 30 thus provides separate pictures the opticalsystem image I(x) and the modified background data images D3(x) andD4(x). As shown in block 40, new point spread functions h_(3T) andh_(4T) have been constructed to account for the combined effect of themethod of sieves and the optical system. These new point spreadfunctions use both the two and three pixel wide point spread functions,h₃ and h₄, previously identified, as well as the optical system PSF h₀,as shown in equation 9(d) and (e), to arrive at the new PSF's.

As shown in block 50, the Richardson-Lucy method is then used toreconstruct the background scene data, D2, with the reconstructed datadefined to be I_(r) (X). FIG. 10 shows an exploded view of theprocessing steps for reconstructing the background scene to obtain thereconstructed background I_(r)(x) .

In FIG. 10 the noise suppressed data D2 from block 20 of FIG. 9B, isused as the first estimate of the true background scene, I_(n)(x)(Module 100). As shown in module 110, this estimate of the truebackground scene is then blurred using the combined method of sieves andoptical system point spread functions to obtain two picturerepresentations I3(x) and I4(x), where I3(x) is given by equation 10(a)and I4(x) is given by equation 10(b). Two new arrays, as shown in module120, are then created by dividing pixel by pixel the noisy scene data,D3(x) and D4(x), by the blurred estimate of the true background scene,I3(x) and I4(x), as shown in equations 10(c) and (d) respectively. Thenew arrays T3(x) and T4(x) are then correlated with the combined methodof sieves and optical system PSF's and the result is multiplied pixel bypixel with the current estimate of the true background scene. In thismanner a new estimate of the true background scene, Z(x), is obtained asshown in module 130 and by equations 10(e)-(g).

The processor then determines whether a predetermined number ofiterations have been performed, as shown in block 140. If thepredetermined number has not been performed, the current estimate of thetrue background is replaced by the new estimate of the true background(Z(x)) and the processing sequence of modules 110-140 are repeated. If,however, the predetermined number has been reached, then the latestestimate of the true background scene is taken to be the reconstructedbackground scenes; that is, I_(r)(x) is taken to equal Z(x) as shown inblock 160. Note that for the optical systems described above thepredetermined number of iterations is preferably between one thousandand two thousand.

In FIG. 9B the reconstructed background I_(r)(x) (equal to Z(x)) is theninput to module 60, where the background subtraction steps are performedto super-resolve the object within the noisy blurred scene data (D1).The reconstructed object, G(x), which has been super-resolved, is thenoutput as shown in module 70.

The background subtraction method for super-resolving the object ofinterest in D1 is detailed in FIG. 11. In FIG. 11 the noisy blurredscene data D1 containing the object to be reconstructed is used asinput, with a low-pass filter (block 600) applied to remove high spatialfrequency noise from the D1 data as shown by equation 11(a). In equation11(a) the transfer function h_(b)(x) represents the Fourier transform ofthe binotf array. The binotf array specifies the non-zero spatialfrequency of the otf located in the fourier planes FIG. 12 shows thefourier transform of an image depicting binotf to ensure no higherfrequencies in the spectrum exist.

As shown in FIG. 12, the values of binotf are 1 up to the optical systemcutoff value f_(c) and 0 beyond that cutoff. The low pass filtered dataD_(f)(x) in FIG. 11 is then multiplied pixel by pixel with the binmaparray (binmap (x)) to separate out a first estimate of the reconstructedobject from the filtered D1 data, as shown in equation 11(b) in module610. The binmap array specifies the region in the scene containing theobject to be super-resolved. The binmap has array elements equal to 1where the object of interest is located and array elements equal to 0everywhere else. FIG. 13 shows a pictorial representation of the binmaparray, where the binmap window (40) holds a region containing an objectof interest (20) consisting of pixels equal to one in the regioncontaining the object and pixels equal to zero everywhere else (30).

The next step in FIG. 11 is to replace the reconstructed backgroundscene pixels, I_(r)(x), by the estimated reconstructed object pixels,D₀(x), at object positions specified by the binmap (40), as shown inmodule 620. Equation 11° provides the mathematical formula for thisreplacement, creating a reconstructed object array S(x). S(x) is thenconvolved with the optical system PSF (h₀) to blur the combination ofthe reconstructed background and the estimated reconstructed object asshown in equation 11(d) of module 630. A new array, N(X), is thencreated in block 640 by dividing, on a pixel by pixel basis, thefiltered D1 scene array (D_(f)) by the blurred combination of thereconstructed background and the estimated reconstructed object (I_(s))as shown in equation 11(e) of module 640. The new array, N(x), is thencorrelated with the optical system PSF (h₀) and multiplied, for eachpixel specified by binmap, by the current estimate of the reconstructedobject. Equation 11(f) of module 650 is then used to determine K(x), thenew estimate of the reconstructed object.

After K(x) has been calculated a check is made, as shown in module 660,to determine whether the specified number of iterations have beencompleted. If more iterations are needed the current estimate of thereconstructed object is replaced by the new estimate as shown byequation 11(g) of module 670. Steps 620-660 are repeated until thespecified number of iterations have been accomplished. When this happensthe latest estimate of the reconstructed objects is taken to be thereconstructed object; that is G(x) is set equal to K(x), as shown inmodule 680 (equation 11H).

In FIG. 9B the reconstructed object G(x) is the output of module 70.This G(x) is in fact the desired super-resolved object. Note that theabove description of the super-resolution method as shown in FIG. 11 isset up to handle non-thinned apertures. For thinned aperture systems,step 630 of FIG. 11 (in which the new scene I_(s)(x) is blurred againusing the optical system's PSF) may be excluded.

Three examples of applying the background reconstruction approach usingthe above-described non-linear technique to obtain super-resolution areillustrated in FIGS. 14-16. FIGS. 14A-D represent simulated bar targetcharts where FIG. 14A represents the truth scene illustrated by a seriesof alternating dark and light bands within a background. FIG. 14Brepresents the blurred image of the truth scene (through a smallaperture), and FIGS. 14C and 14D represent the reconstructed imagesusing the non-linear background reconstruction method previouslydescribed. FIGS. 15A-D are associated respectively with FIGS. 14A-D andrepresent a one-dimensional Fourier transform cut through each of the“scenes”, thus clearly illustrating the spatial frequencies that havebeen restored to the reconstructed image.

FIGS. 16A-C represent the application of the non-linear method to athinned aperture system. In this case, the thinned apertureconfiguration is an annulus. It should be noted, however, that themethod may be utilized with any thinned aperture design. FIG. 16Arepresents a computer generated ground scene (i.e. the truth scene). Theblurred image of that scene is then depicted in FIG. 16B, while thefinal reconstructed, super-resolved image is shown in FIG. 16C.

FIGS. 17A-C represent images of figures taken from a CCD camera. FIG.17A represents the truth scene (a picture of a toy spacemen), while FIG.17B shows the blurred image of the scene (observed through a smallaperture). FIG. 17C represents the reconstructed image, and FIG. 17Dshows the magnitude of the difference of the two-dimensional Fouriertransform between the truth scene in FIG. 17A and the blurred image ofFIG. 17B. FIG. 17E shows the difference between the truth scene and thefirst stage of reconstruction (i.e. the deconvolved figure), while FIG.17F shows the magnitude of the difference of the two-dimensional Fouriertransform and the truth scene for the reconstructed, super resolvedimage. Note that black indicates a 0 difference, which is the desiredresult, while white indicates a maximum difference. As one can see froma comparison of FIGS. 17D, E and F, the radius of FIG. 17D correspondsto the cutoff of the optical system or camera, and the deconvolved imagefrequencies in FIG. 17E have been enhanced inside the cutoff but remainzero outside the cutoff. The super-resolved figure in FIG. 17F hasfurther improved the image by restoring frequencies outside the cutoffas can be shown by the increased blackness of the figure with respect toeither FIGS. 17D or 17E. This is a clear demonstration that superresolution has occurred. FIGS. 18A-B represent a graphical illustrationof the truth, blurred, and reconstructed super-resolved images for SNRvalues of 50 and 100 respectively. FIGS. 18A and B show that thenon-linearly reconstructed images closely parallel the truth images.

In an alternative embodiment, the reconstruction approach using a lineartransformed method is now described. When reconstructing either thebackground scene or the localized object, the imaging system ismathematically characterized by a linear operator represented by amatrix. To restore either a background scene or the localized object, aninverse imaging matrix corresponding to the inverse operator must beconstructed. However, due to the existence of system noise, applying theinverse imaging matrix to the image is intrinsically unstable andresults in a poorly reconstructed image. However by applying aconstrained least squares procedure such as Tikhonov regularization, aregularized pseudo-inverse (RPI) matrix may be generated. Zero-orderTikhonov regularization is preferably used, although higher orderTikhonov regularization may sometimes give better results. The detailsof this are not described here, as Tikhonov regularization is well-knownin the art.

A key quantity used to construct the RPI matrix is the regularizationparameter which controls the image restoration. Larger parameter valuesprotect the restored image from the corrupting effects of the opticalsystem but result in a restored image which has lower resolution. Anoptimum or near optimum value for the regularization parameter may bederived automatically from the image data. Singular value decomposition(SVD) of the imaging operator matrix may be used to compute the RPImatrix. By estimating the noise or error level of the degraded image thesingular values of the matrix determine the extent to which the fullinformation in the original scene may be recovered. Note that the use ofthe SVD process is not essential in determining the RPI matrix, andother methods such as QR decomposition of the imaging matrix may also beused to achieve essentially the same result.

The imaging matrix size increases approximately as the square of theimage size, and the computational burden associated with forming the RPIof the imaging matrix quickly becomes intolerable. However, in thespecial case that the image and object fields are the same size and aresampled at the same intervals (as is the case here), the imaging matrixcan be expanded into circulant form by inserting appropriatelypositioned, additional columns. A fundamental theorem of matrix algebrais that the Fourier transform diagonalizes a circulant matrix. Thisallows the reduction of the image reconstruction algorithm to a pair ofone dimensional fast Fourier transforms, followed by a vector—vectormultiplication, and finally an inverse one dimensional fast Fouriertransform. This procedure allows the image restoration by this Tikhonovregularization technique to be done entirely in the Fourier transformdomain, dramatically reducing the time required to compute thereconstructed image. FIG. 19 provides an illustration of the steps takento obtain the reconstructed image using the linear transform method.

A flow diagram of the Linear Algebra Technique according to the presentinvention is shown in FIG. 19. Referring now to FIG. 19, the techniquefirst converts the imaging data collected by the optical system of thesensor into a matrix of the form g_(l)(i, j)=Σ_(m)Σ_(n)h(I−m, j−n)f(m,n)+n₁(i, j), where h is a matrix representation of the point spreadfunction of the optical system and f is the matrix representation of theunblurred background with the embedded object, while n₁ is the matrixrepresentation of the additive white noise associated with the imagingsystem (module 12). Next, imaging data g₂ comprising the image scenedata which contains only the background data is obtained in the form ofg₂(i, j)−Σ_(m)Σ_(n)h (I−m, j−n)b(m,n)+n₂ (i,j) where b is a matrixrepresentation of the unblurred background data taken alone and n₂ is amatrix representation of additive system white noise (module 14). Bothg₁ and g₂ are then low-pass filtered to the cut-off frequency of theoptical system to reduce the effects of noise (module 15). Module 16then shows the subtraction step whereby the matrix representation (g₃)of the difference between blurred scene data containing the backgroundand object of interest (g₁) and the blurred scene containing only thebackground data (g₂) is formed as (g₁−g₂). Next, the position and sizeof the object of interest is specified by choosing x,y coordinatesassociated with image matrix (g₁−g₂) (module 18). A segment ofsufficient size to contain the blurred object in its entirety is thenextracted from the matrix representation of (g₁−g₂), as shown in Module20. That is, an area equal to the true extent of the local object plusits diffracted energy is determined An identically located segment (i.e.segment having the same x,y coordinates) is extracted from the blurredbackground scene matrix g₂ as shown in module 22. The two image segmentsoutput from module 15 and 20 are then input to module 24 to restore g₂and (g₁−g₂) using nth order Tikhonov regularization. The restoredsegments are then added together as shown in step 26 and the areacontaining the restored object of interest is extracted therefrom, asshown in module 28.

The resulting reconstructed image includes much of the spatialresolution which was lost due to diffraction blurring effects.

It should be noted that the present invention is not limited to any onetype of optical sensor device. The principles which have been describedherein apply to many types of applications which include, but are notlimited to, Optical Earth Resource Observation Systems (both Air andSpaceborne), Optical Weather Sensors (both Air and Spaceborne), TerrainMapping Sensors (both Air and Spaceborne), Surveillance Sensors (bothAir and Spaceborne), Optical Phenomenology Systems (both Air andSpaceborne), Imaging Systems that utilize optical fibers such as MedicalProbes, Commercial Optical Systems such as Television Cameras,Telescopes utilized for astronomy and Optical Systems utilized forPolice and Rescue Work.

While the invention has been particularly shown and described withreference to preferred embodiments thereof, it will be understood bythose skilled in the art that changes in form and details may be madetherein without departing from the spirit and scope of the presentinvention.

What is claimed is:
 1. In an optical system having a detector means andprocessor means in which imaging data is obtained comprising noisyblurred scene data containing an object to be reconstructed, and noisyblurred background data of the same scene, a method for increasing thespatial resolution of the imaging data produced by the optical systemfor providing an image of higher resolution, comprising the steps of:converting the imaging data into a first matrix; regularizing the firstmatrix by performing nth order Tikhonov regularization to the firstmatrix to provide a regularized pseudo-inverse (RPI) matrix; andapplying the RPI matrix to the first matrix to provide a reconstructedimage of the object.
 2. The method of claim 1 wherein the optical systemis diffraction limited.
 3. The method according to claim 1, wherein theRPI matrix is determined via a singular value decomposition (SVD) of thefirst matrix obtained from the point spread function of the opticalsystem.
 4. The method according to claim 1, wherein the RPI matrix isdetermined using QR decomposition of the first matrix.
 5. A method forsubstantially increasing the spatial resolution of imaging data producedby an optical system, comprising the steps of: converting the imagingdata into a first matrix g₁, comprising background data and object ofinterest data, and a second matrix g₂ comprising background data;subtracting the second matrix g₂ from the first matrix g₁ to obtain athird matrix g₃ indicative of the difference between the first andsecond matrices, specifying a position and size of an object of interestin the third matrix g₃; extracting from the third matrix g₃ a segmenthaving coordinates which completely include a blurred version of theobject of interest; performing nth order Tikhonov regularization on eachsuch extracted segment and said second matrix to restore the segment,adding such restored segments, and extracting an area of the addedrestored segments containing an object of interest to obtain restoredimage data containing a restored object of interest.
 6. The methodaccording to claim 5, wherein the first matrix comprises an image scenecontaining the object of interest and background data, and wherein thesecond matrix comprises the image scene containing only the backgrounddata.
 7. The method according to claim 5, further comprising the step oflow pass filtering the first and second matrices to the system cutofffrequency.
 8. The method according to claim 6, wherein the step ofextracting a segment having coordinates which completely include ablurred version of an object of interest comprises extracting an areaequal to the extent of such object of interest plus the diffractedenergy associated with the object of interest.
 9. The method of claim 5wherein the optical system is diffraction limited.
 10. The method ofclaim 8 wherein the optical system is diffraction limited.
 11. In anoptical system having a detector means and processor means in whichimage data is obtained comprising noisy blurred scene data containing anobject to be reconstructed, and noisy blurred background data of thesame scene, an apparatus for increasing the spatial resolution of theimaging data produced by the optical system for providing an image ofhigher resolution, comprising: means for converting the imaging datainto a first matrix; means for regularizing the first matrix byperforming nth order Tikhonov regularization to the first matrix toprovide a regularized pseudo-inverse (RPI) matrix; and means forapplying the RPI matrix to the first matrix to provide a reconstructedimage of said object.
 12. The apparatus of claim 11 wherein the opticalsystem is diffraction limited.
 13. The apparatus of claim 11 wherein theoptical system has a numerical aperture and the detector means has atleast five detectors spread across the central lobe of the diffractionpattern determined by the numerical aperture.
 14. The apparatus of claim11, wherein the RPI matrix is determined via a singular valuedecomposition (SVD) of the first matrix obtained from the point spreadfunction of the optical system.
 15. The apparatus of claim 11, whereinthe RPI matrix is determined using QR decomposition of the first matrix.16. An apparatus for substantially increasing the spatial resolution ofimaging data produced by an optical system comprising: means forconverting the imaging data into a first matrix g₁ comprising backgrounddata and object of interest data, and a second matrix g₂ comprisingbackground data; means for subtracting the second matrix g₂ from thefirst matrix g₁ to obtain a third matrix g₃ indicative of the differencebetween said first and second matrices; means for specifying a positionand a size of an object of interest in the third matrix g₃; means forextracting from the third matrix at least a segment having coordinateswhich completely include a blurred version of the object of interest;and means for performing nth order Tikhonov regularization on each suchextracted segment and said second matrix to restore the segment, addingsaid restored segments, and extracting an area of added restoredsegments containing the object of interest to obtain the restored imagedata containing a restored object of interest.
 17. The apparatus ofclaim 16 wherein the optical system is diffraction limited.
 18. Theapparatus of claim 16 wherein the optical system has a numericalaperture and the detector means has at least five detectors spreadacross the central lobe of the diffraction pattern determined by thenumerical aperture.
 19. The apparatus of claim 16, wherein the firstmatrix comprises an image scene containing an object of interest andbackground data, and wherein the second matrix comprises the image scenecontaining only background data.
 20. The apparatus of claim 19, furthercomprising means for low pass filtering the first and second matrices tothe system cutoff frequency.
 21. The apparatus of claim 19, wherein themeans for extracting at least a segment having coordinates whichcompletely include a blurred version of the object of interest comprisesmeans for extracting an area equal to the extent of the object ofinterest plus the diffracted energy associated with the object ofinterest.